Illumination Analysis for Synthetic/Natural Hybrid Image Sequence Generation

For virtual studio TV production, multi-site video conference applications using a virtual meeting room and for synthetic/natural hybrid coding according to the forthcoming ISO/MPEG-4 standard, a synthetic and a natural scene are mixed to generate a synthetic/natural hybrid image sequence. For realism, the illumination in both scenes should be identical. In this paper, the illumination of the natural scene is estimated automatically and applied to the synthetic scene. For illumination estimation, moving real objects are segmented in the natural image sequence and 3D modeled. The 3D shape, 3D motion and the displaced frame difference between two succeeding images are evaluated to estimate three illumination parameters. The parameters describe a distant point light source and ambient light. Using the estimated illumination parameters, the synthetic scene is rendered and mixed to the natural image sequence. Experimental results with a moving virtual ball mixed automatically into a real video telephone sequence show that virtual objects appear naturally having the same shading and shadows as the real person.